On Versatile Video Coding at UHD with Machine-Learning-Based Super-Resolution

Coding 4K data has become of vital interest in recent years, since the amount of 4K data is significantly increasing. We propose a coding chain with spatial down- and upscaling that combines the next-generation VVC codec with machine learning based single image super-resolution algorithms for 4K. The investigated coding chain, which spatially downscales the 4K data before coding, shows superior quality than the conventional VVC reference software for low bitrate scenarios. Throughout several tests, we find that up to 12 % and 18 % Bjontegaard delta rate gains can be achieved on average when coding 4K sequences with VVC and QP values above 34 and 42, respectively. Additionally, the investigated scenario with up- and downscaling helps to reduce the loss of details and compression artifacts, as it is shown in a visual example.Comment: Originally published as conference paper at QoMEX 202

A Bit Stream Feature-Based Energy Estimator for HEVC Software Encoding

The total energy consumption of today's video coding systems is globally significant and emphasizes the need for sustainable video coder applications. To develop such sustainable video coders, the knowledge of the energy consumption of state-of-the-art video coders is necessary. For that purpose, we need a dedicated setup that measures the energy of the encoding and decoding system. However, such measurements are costly and laborious. To this end, this paper presents an energy estimator that uses a subset of bit stream features to accurately estimate the energy consumption of the HEVC software encoding process. The proposed model reaches a mean estimation error of 4.88% when averaged over presets of the x265 encoder implementation. The results from this work help to identify properties of encoding energy-saving bit streams and, in turn, are useful for developing new energy-efficient video coding algorithms.Comment: arXiv admin note: text overlap with arXiv:2207.0267

Sweet Streams are Made of This: The System Engineer's View on Energy Efficiency in Video Communications

In recent years, the global use of online video services has increased rapidly. Today, a manifold of applications, such as video streaming, video conferencing, live broadcasting, and social networks, make use of this technology. A recent study found that the development and the success of these services had as a consequence that, nowadays, more than 1% of the global greenhouse-gas emissions are related to online video, with growth rates close to 10% per year. This article reviews the latest findings concerning energy consumption of online video from the system engineer's perspective, where the system engineer is the designer and operator of a typical online video service. We discuss all relevant energy sinks, highlight dependencies with quality-of-service variables as well as video properties, review energy consumption models for different devices from the literature, and aggregate these existing models into a global model for the overall energy consumption of a generic online video service. Analyzing this model and its implications, we find that end-user devices and video encoding have the largest potential for energy savings. Finally, we provide an overview of recent advances in energy efficiency improvement for video streaming and propose future research directions for energy-efficient video streaming services.Comment: 16 pages, 5 figures, accepted for IEEE Circuits and Systems Magazin

Learning Frequency-Specific Quantization Scaling in VVC for Standard-Compliant Task-driven Image Coding

Today, visual data is often analyzed by a neural network without any human being involved, which demands for specialized codecs. For standard-compliant codec adaptations towards certain information sinks, HEVC or VVC provide the possibility of frequency-specific quantization with scaling lists. This is a well-known method for the human visual system, where scaling lists are derived from psycho-visual models. In this work, we employ scaling lists when performing VVC intra coding for neural networks as information sink. To this end, we propose a novel data-driven method to obtain optimal scaling lists for arbitrary neural networks. Experiments with Mask R-CNN as information sink reveal that coding the Cityscapes dataset with the proposed scaling lists result in peak bitrate savings of 8.9 % over VVC with constant quantization. By that, our approach also outperforms scaling lists optimized for the human visual system. The generated scaling lists can be found under https://github.com/FAU-LMS/VCM_scaling_lists.Comment: Originally submitted at IEEE ICIP 202

Component-wise Power Estimation of Electrical Devices Using Thermal Imaging

This paper presents a novel method to estimate the power consumption of distinct active components on an electronic carrier board by using thermal imaging. The components and the board can be made of heterogeneous material such as plastic, coated microchips, and metal bonds or wires, where a special coating for high emissivity is not required. The thermal images are recorded when the components on the board are dissipating power. In order to enable reliable estimates, a segmentation of the thermal image must be available that can be obtained by manual labeling, object detection methods, or exploiting layout information. Evaluations show that with low-resolution consumer infrared cameras and dissipated powers larger than 300mW, mean estimation errors of 10% can be achieved.Comment: 10 pages, 8 figure

Power Reduction Opportunities on End-User Devices in Quality-Steady Video Streaming

This paper uses a crowdsourced dataset of online video streaming sessions to investigate opportunities to reduce the power consumption while considering QoE. For this, we base our work on prior studies which model both the end-user's QoE and the end-user device's power consumption with the help of high-level video features such as the bitrate, the frame rate, and the resolution. On top of existing research, which focused on reducing the power consumption at the same QoE optimizing video parameters, we investigate potential power savings by other means such as using a different playback device, a different codec, or a predefined maximum quality level. We find that based on the power consumption of the streaming sessions from the crowdsourcing dataset, devices could save more than 55% of power if all participants adhere to low-power settings.Comment: 4 pages, 3 figure

Processing Energy Modeling for Neural Network Based Image Compression

Nowadays, the compression performance of neural-networkbased image compression algorithms outperforms state-of-the-art compression approaches such as JPEG or HEIC-based image compression. Unfortunately, most neural-network based compression methods are executed on GPUs and consume a high amount of energy during execution. Therefore, this paper performs an in-depth analysis on the energy consumption of state-of-the-art neural-network based compression methods on a GPU and show that the energy consumption of compression networks can be estimated using the image size with mean estimation errors of less than 7%. Finally, using a correlation analysis, we find that the number of operations per pixel is the main driving force for energy consumption and deduce that the network layers up to the second downsampling step are consuming most energy.Comment: 5 pages, 3 figures, accepted for IEEE International Conference on Image Processing (ICIP) 202

Extended Signaling Methods for Reduced Video Decoder Power Consumption Using Green Metadata

In this paper, we discuss one aspect of the latest MPEG standard edition on energy-efficient media consumption, also known as Green Metadata (ISO/IEC 232001-11), which is the interactive signaling for remote decoder-power reduction for peer-to-peer video conferencing. In this scenario, the receiver of a video, e.g., a battery-driven portable device, can send a dedicated request to the sender which asks for a video bitstream representation that is less complex to decode and process. Consequently, the receiver saves energy and extends operating times. We provide an overview on latest studies from the literature dealing with energy-saving aspects, which motivate the extension of the legacy Green Metadata standard. Furthermore, we explain the newly introduced syntax elements and verify their effectiveness by performing dedicated experiments. We show that the integration of these syntax elements can lead to dynamic energy savings of up to 90% for software video decoding and 80% for hardware video decoding, respectively.Comment: 5 pages, 2 figure